On the optimality of universal classifiers for finite-length individual test sequences

On the optimality of universal classifiers for finite-length individual test sequences. Abstract It has been demonstrated that if two individual sequences are independent realizations of two finite-order, finite alphabet, stationary Markov processes, an empirical divergence measure (ZMM) that is based on cross-parsing of one sequence relative to the second one converges to the relative entropy almost surely. This leads to a realization of an empirical, linear complexity universal classifier which is asymptotically optimal in the sense that the probability of classification error vanishes as the length of the sequence tends to infinity if the KL-divergence between the two processes is positive. It is demonstrated that a version of the ZMM is not only asymptotically optimal as the length of the sequences tends to infinity, but is also essentially-optimal for a class of finite-length sequences that are realizations of finite-alphabet, vanishing memory processes with positive transitions in the sense that the probability of classification error vanishes if the length of the sequences is larger than some positive integer No and leads to an asymptotically optimal classification algorithm. At the same time no universal classifier can yield an efficient discrimination between any two distinct processes in this class, if the length of the two sequences N is such that log N < log N o , even if the KL-divergence between the two processes is positive. It is further demonstrated that not every asymptotically optimal universal classification algorithm is also essentially optimal. A variable length (VL) divergence that converges to the KL-divergence when the length of the sequences tends to infinity, is defined. Another universal classification algorithm which, like ZMM is also based on cross-parsing, is shown to be optimal relative to the VL divergence (rather than being just essentially optimal) for any two finite-length sequences that are realizations of vanishing-memory processes.